# Masked Language Modeling

A Japanese large language model based on the modernBERT-base architecture, supporting a maximum sequence length of 8192, trained on 3.4TB of Japanese corpus

DUO is a pre-trained model for text generation, which can be used for masked language modeling tasks. It is trained on the OpenWebText corpus and has good performance.

A French language model pre-trained on 1 trillion high-quality French texts, the French version of ModernBERT

DUO is a pretrained model based on the Transformers library, focusing on masked language modeling tasks and suitable for the field of natural language processing.

CamemBERTv2 is a French language model pre-trained on a 275 billion-word French text corpus, serving as the second-generation version of CamemBERT. It adopts the RoBERTa architecture with optimized tokenizer and training data.

A Kazakh language model based on the RoBERTa architecture, trained from scratch using RobertaForMaskedLM, suitable for Kazakh text processing tasks.

CodeBERTa is a code understanding model based on the RoBERTa architecture, specifically trained for multiple programming languages, capable of efficiently handling code-related tasks.

SaudiBERT is the first large-scale pre-trained language model specifically focused on Saudi dialect text, trained on a massive corpus of Saudi tweets and forum posts.

A multilingual ALBERT model pretrained with masked language modeling (MLM) objective, supporting 64k vocabulary size and case sensitivity

Indus is an encoder-only Transformer model based on RoBERTa, specifically optimized for NASA Science Mission Directorate (SMD) applications, suitable for scientific information retrieval and intelligent search.

Albertina 100M PTBR is a foundational large language model for Brazilian Portuguese, belonging to the BERT family of encoders, based on the Transformer neural network architecture, and developed on the DeBERTa model.

Albertina 100M PTPT is a foundational large language model for European Portuguese (Portugal), belonging to the BERT family of encoders. It is based on the Transformer neural network architecture and developed upon the DeBERTa model.

A medium-sized BERT language model using Masked Language Modeling (MLM) as the pre-training objective.

Model fine-tuned based on distilroberta-base, specific purpose not clearly stated

ESM-2 is a state-of-the-art protein model trained with masked language modeling objectives, suitable for fine-tuning on various protein sequence tasks.

FinBERT is a BERT model pretrained on financial communication texts, specifically designed for financial natural language processing tasks.

K-12BERT is a BERT model obtained through continuous pretraining on K-12 basic education data, optimized specifically for educational scenarios

A language model specifically designed for astronomy and astrophysics, developed by the NASA/ADS team, supporting masked token filling, named entity recognition, and text classification tasks.

This model is based on albert-base-v2's masked language model, specifically fine-tuned for electronic product attribute correction tasks

A large Japanese RoBERTa model pretrained on Japanese Wikipedia and the Japanese portion of CC-100, suitable for Japanese natural language processing tasks.

A RoBERTa model pretrained on English legal and administrative texts, specializing in language understanding tasks in the legal domain

A BERT large model pretrained on English legal and administrative texts using RoBERTa pretraining objectives

A RoBERTa model employing pre-layer normalization technology, studying the impact of masking ratio in masked language modeling

TOD-XLMR is a multilingual task-oriented dialogue model developed based on XLM-RoBERTa, employing a dual-objective joint training strategy to enhance dialogue understanding capabilities

SKEP (Sentiment Knowledge Enhanced Pre-training) was proposed by Baidu in 2020, specifically designed for sentiment analysis tasks. The model incorporates multi-type knowledge through sentiment masking techniques and three sentiment pre-training objectives.

A masked language model (MLM) fine-tuned based on bert-base-uncased, suitable for text filling tasks

BERT is a transformer model pre-trained on large-scale English corpora through self-supervised learning, utilizing masked language modeling and next sentence prediction objectives

This model freezes all pre-trained BERT weights except the last layer, focusing on masked language modeling tasks to explore the model's generalization ability in processing quantitative statements.

A BERT base model pretrained on Indonesian Wikipedia using Masked Language Modeling (MLM) objective, case insensitive.

A Japanese RoBERTa-based pretrained model, trained on Japanese Wikipedia and the Japanese portion of CC-100.

TAPAS is a Transformer-based table question answering model pre-trained in a self-supervised manner on Wikipedia tables and associated text, supporting table understanding and question answering tasks.

This is a dummy RoBERTa model for unit testing and continuous integration, primarily used for demonstration and testing purposes

TAPAS (Table Parsing) is a pre-trained language model developed by Google specifically for handling table-related tasks.

PolitBERT is a BERT model specifically designed for analyzing speeches, interviews, and press conference content of political figures in English-speaking countries.

TAPAS is a table-based pretrained language model specifically designed for tasks related to tabular data.

BERT is a pre-trained language model based on the Transformer architecture, trained on English corpus through masked language modeling and next sentence prediction tasks.

A Chinese RoBERTa model pretrained on CLUECorpusSmall, with 8 layers and 512 hidden units, suitable for various Chinese NLP tasks.

Arabic pretrained version of ALBERT large model, trained on approximately 4.4 billion words of Arabic corpus

TAPAS is a Transformer-based table question answering model pre-trained in a self-supervised manner on English Wikipedia table data, supporting table QA and entailment tasks.

A large-scale multi-task representation model achieved through pre-finetuning, based on the RoBERTa architecture, outperforming the original roberta-base on GLUE and question answering tasks